Implement complete database foundation for pipeline-based job tracking and monitoring:

Database Tables:
• pipelines - High-level workflow grouping with correlation IDs for end-to-end tracing
• job_runs - Individual job execution tracking with full lifecycle management
• job_dependencies - Workflow orchestration with success/completion dependency types
• job_metrics - Detailed performance metrics (CPU, memory, execution time, business metrics)
• variant_annotation_status - Granular variant-level annotation tracking with success data

Key Features:
• Pipeline workflow management with dependency resolution
• Comprehensive job lifecycle tracking (pending → running → completed/failed)
• Retry logic with configurable limits and backoff strategies
• Resource usage and performance metrics collection
• Variant-level annotation status for debugging failures
• Correlation ID support for request tracing across system
• JSONB metadata fields for flexible job-specific data
• Optimized indexes for common query patterns

Schema Design:
• Foreign key relationships maintain data integrity
• Check constraints ensure valid enum values and positive numbers
• Strategic indexes optimize dependency resolution and metrics queries
• Cascade deletes prevent orphaned records
• Version tracking for audit and debugging

Models & Enums:
• SQLAlchemy models with proper relationships and hybrid properties
• Comprehensive enum definitions for job/pipeline status and failure categories

…pings

…ods and failures

…atures

Add comprehensive job lifecycle management with status-based completion:

* Implement convenience methods for common job outcomes:
  - succeed_job() for successful completion
  - fail_job() for error handling with exception details
  - cancel_job() for user/system cancellation
  - skip_job() for conditional job skipping
* Enhance progress tracking with increment_progress() and set_progress_total()
* Add comprehensive error handling with specific exception types
* Improve job state validation and atomic transaction handling
* Implement extensive test coverage for all job operations

- Created PipelineManager capable of coordinating jobs within a pipeline context
- Introduced `construct_bulk_cancellation_result` to standardize cancellation result structures.
- Added `job_dependency_is_met` to check job dependencies based on their types and statuses.
- Created comprehensive tests for PipelineManager covering initialization, job coordination, status transitions, and error handling.
- Implemented mocks for database and Redis dependencies to isolate tests.
- Added tests for job enqueuing, cancellation, pausing, unpausing, and retrying functionalities.

Adds decorators for managed jobs and pipelines. These can be applied to async ARQ functions to automatically persist their state as they execute

…ation and error handling

…b management

…job management

…d job management

…sion process and enhance logging

… enum to str inheritance

…ance test coverage

…gnomad tests

…year and month parameters

…ll tests on main branch

…calls

Implements 24-hour Redis cache for ClinGen Allele Registry API responses,
significantly reducing API load when processing multiple ClinVar control
versions that query the same alleles. Converts three ClinGen functions to
async with @cached decorator, implements memory backend for testing, and
handles 404 responses as cacheable "no data" results while raising exceptions
for other API failures. Includes comprehensive test coverage and type stubs
for the untyped aiocache library.

- Add aiocache optional dependency with Redis backend support
- Create cache configuration module with environment-based backend selection
- Convert get_canonical_pa_ids, get_matching_registered_ca_ids, and
  get_associated_clinvar_allele_id to async cached functions
- Return empty string/list for "no data" cases to enable caching of modal outcomes
- Implement 404-specific error handling: cache permanent absences, raise for transient failures
- Add memory cache backend for testing without Redis dependency
- Create type stubs for aiocache.Cache and aiocache.cached decorator
- Add 43 new tests covering caching behavior, configuration, and network interactions

… better transaction control

Add periodic cleanup job to detect and recover jobs stuck in QUEUED,
RUNNING, or PENDING states beyond timeout thresholds. Jobs can become
stalled due to worker crashes, race conditions during enqueue, network
issues, or database transaction failures.

Cleanup logic:
- QUEUED jobs stalled >10 min (stuck between state change and ARQ pickup)
- RUNNING jobs stalled >60 min (worker likely crashed mid-execution)
- PENDING jobs stalled >30 min (pipeline coordination failure)

Unified retry handler workflow:
1. Fail job with TIMEOUT category for being stalled
2. Check retry eligibility via should_retry()
3. If eligible: prepare retry and check dependencies
4. For pipeline jobs: validate dependencies before enqueueing
   - Skip if dependencies failed (leave in PENDING for pipeline manager)
   - Wait if dependencies not ready (leave in PENDING)
   - Enqueue if dependencies satisfied
5. If max retries exceeded or enqueue fails: mark SYSTEM_ERROR

Key features:
- Graceful handling of edge cases (missing started_at, max retries)
- Pipeline dependency awareness (avoids enqueueing guaranteed failures)
- Comprehensive test coverage (42 tests: 22 unit, 19 integration, 1 ARQ)

This safety net ensures jobs don't remain in limbo indefinitely and
provides automatic recovery from transient infrastructure failures.

The `hgvs_assay_level` field was not being set during the mapping
job, deferring it for a later job. Derive it from the post-mapped VRS
object at creation time using `get_hgvs_from_post_mapped`.

Adds a test assertion to verify the field is populated after mapping.

The previous behavior always scoped retirement of current annotation
records to the same (variant, type, version) tuple. This is correct
for ClinVar, where records from different source
versions should coexist independently.

For the mapping pipeline, ClinGen and gnomAD,, a new run should supersede all prior
results regardless of which version produced them.

- Add `replace_all_versions: bool = True` parameter to
  `AnnotationStatusManager.add_annotation`; default clobbers all
  versions so existing callers that omit the flag get the broader
  retirement behavior
- Explicitly pass `replace_all_versions=False` on all ClinVar call
  sites to preserve per-version record coexistence
- Add `TestAnnotationStatusManagerReplaceAllVersionsUnit` covering
  both modes across type, variant, and version isolation boundaries

…failures

… results